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material. 

yC  Citations  of  commercial  organizations  and  trade  names  in 
this  report  do  not  constitute  an  official  Department  of  Ay 
endorsement  or  approval  of  the  products  or  services  of  these 
organizations . 

In  conducting  research  using  animals,  the  investigator (s) 
idhired  to  the  "Guide  for  the  Care  and  Dse  of  Lab^^^^^ 

^Animals,"  prepared  by  the  Committee  on  Care  .and  Use  pf  Lab^  V 

Animals  of  the  Institute  of  Laboratory  Resources,  Nationa 
Research  Council  (NIH  Publication  No.  86-23,  Revised  1985). 

yk  For  the  protection  of  human  subjects,  the  investigator (s) 
idhered  to  policies  of  applicable  Federal  Law  45  CFR  46. 

In  conducting  research  utilizing  recombinant .DNA  technQlpgy, 
the^investigator (s)  adhered  to  current  guidelines  promulgated  by 
the  National  Institutes  of  Health. 

. ::  In  the  conduct  of  research  utilizing  recoi^inant  "DNA/  the 

iil^stigator.C  the  NIH  Guidelines  for  .Research 

Involving  Recombinant  DNA  Molecules. 

In  the  conduct  of  research  involving  hazardous  :opani sms, 
■H^investigator  ( s )  adhered  to  the  CDC-NIH  Guide  for  Hiosaf ety 
Microbiological  and  Biomedical  Laboratories 
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INTRODUCTION 


We  have  developed  a  stereoscopic  mammography  system  using  state-of-the-art  digital 
technology  that  enables  a  mammographer  to  view  the  internal  structure  of  the  breast  directly  in 
depth.  Stereo  mammograms  are  obtained  by  taking  two  x-rays  of  the  breast  from  viewpoints 
separated  by  about  6  degrees,  each  view  captured  by  a  digital  CCD  camera.  The  two  digital 
images  are  viewed  on  a  stereo  display  workstation  by  a  radiologist  wearing  stereo-viewing  eye¬ 
glasses.  The  system  provides  the  user  with  control  over  display  parameters  such  as  brightness, 
contrast,  grayscale  polarity,  and  inversion  of  depth. 

The  primary  aims  of  this  project  are  to  further  develop  and  refine  this  system  and  then  to 
evaluate  its  effectiveness  in  increasing  the  accuracy  of  diagnosis  of  breast  cancer  relative  to  the 
diagnostic  accuracy  afforded  by  standard,  non-stereo  film  views.  The  expected  increase  in 
diagnostic  accuracy  could  increase  the  yield  of  biopsy  substantially  by  allowing  the  radiologist  to 
more  confidently  recommend  accelerated  follow-up  of  truly  benign  disease  and  biopsy  of  truly 
malignant  disease. 

One  of  two  final  goals  in  this  project  is  to  conduct  a  reading  accuracy  study  with  a  group 
of  mammographers.  Each  will  rate  the  probability  of  malignancy  for  each  of  a  set  of  cases 
acquired  during  the  project,  first  reading  only  the  conventional  planar  films,  then  reading  the 
films  supplemented  by  non-stereo  viewing  of  the  digital  mammograms,  and  finally  reading  the 
films  supplemented  by  stereo  viewing  of  the  digital  mammograms.  We  will  use  ROC  methods 
(Swets,  1979,  1986a,  1988)  to  evaluate  the  increase  in  diagnostic  accuracy  due  to  the  addition  of 
the  digital  non-stereo  views,  and  the  increase  due  to  addition  of  the  stereoscopic  view. 

Our  second  goal  is  to  determine  the  further  improved  accuracy  that  is  attainable  by  use  of 
a  feature-based  aiding  system.  In  a  second,  separate  study,  readers  will  read  each  case  and  assign 
a  numerical  value  to  each  of  a  set  of  stereo  and  non-stereo  mammographic  features  on  a 
checklist.  A  statistical  prediction  rule  will  be  trained  on  those  feature  sets  and  evaluated,  using 
ROC  methods,  as  a  diagnostic  aid  to  further  increase  the  accuracy  attained  by  stereo 
mammography  (Getty  et  al.,  1988;  Seltzer  et  al.,  1996;  Getty  et  al.,  1997). 

Three  preliminary  steps  are  necessary  before  conducting  these  studies:  (1)  to  further 
develop  and  refine  the  user  control  interface  of  the  stereo  display  system,  (2)  to  acquire  a  large 
case  set  of  standard  films  and  digital  stereo  mammograms  from  consenting  patients  about  to 
undergo  breast  biopsy,  and  (3)  to  identify  new  stereo  features,  and  features  that  may  be  seen 
better  in  stereo,  with  the  aid  of  expert  mammographers  and  to  develop  quantitative  rating  scales 
for  each. 

Advantages  of  stereo  mammography  over  standard  non-stereo  film  mammography 

Mammography  is  known  to  be  one  of  the  most  difficult  and  demanding  of  the  image- 
based  examinations  conducted  by  radiologists.  Focal  abnormalities  are  often  difficult  to  detect 
because  of  obscuration  by  superimposed  normal  parenchymal  tissue,  particularly  in  the  dense 
breast.  Moreover,  once  detected,  a  focal  abnormality  is  often  difficult  to  diagnose  —  first  because 
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of  the  difficulty  of  separating  it  perceptually  from  superimposed  parenchymal  tissue,  and  second 
because  of  the  subtlety,  even  if  unobscured,  of  the  visual  features  distinguishing  benign  and 
malignant  lesions.  We  believe  that  stereo  mammography  will  significantly  help  with  both  of 
these  problems,  by  providing  the  mammographer  with  much  more  precise  information  about  the 
three-dimensional  properties  of  each  abnormality  and  by  visually  segregating  the  abnormality 
from  the  parenchymal  context  in  which  it  resides. 

With  a  conventional  two-view  mammographic  study,  the  volumetric  locus  and  the 
volumetric  properties  of  a  focal  abnormality  can  only  be  determined  through  a  rather  demanding 
and  imprecise  cognitive  process.  The  abnormality  must  first  be  localized,  and  its  two- 
dimensional,  planar  properties  assessed  separately  in  two  views.  Then,  to  determine  its 
volumetric  locus  and  volumetric  properties,  the  two  sets  of  planar  information  have  to  be  merged 
cognitively,  into  what  can  only  be  a  rough  impression  of  where  the  abnormality  lies  and  how  it 
would  look  if  it  could  be  seen  in  the  volume.  With  the  stereo  display,  the  mammographer  does 
not  have  to  rely  on  such  a  demanding  and  imprecise  process.  He/she  can  look  directly  into  the 
volume  and,  via  a  very  natural  and  effortless  perceptual  process,  see  precisely  where  the 
abnormality  is  and  what  it  looks  like.  Moreover,  the  perception  is  a  global  one  that  can  reveal  in 
a  rich  and  precise  way  how  the  abnormality  relates  to  the  surrounding  parenchymal  structure  as 
well  as  to  other  possible  focal  abnormalities.  These  gains  in  the  richness,  precision  and  global 
nature  of  the  volumetric  information  that  can  be  obtained  from  a  stereo  display  have  implications 
for  potentially  large  improvement  both  in  the  detection  and  diagnosis  of  focal  abnormalities.  In 
this  project,  the  focus  is  on  improvements  in  accuracy  of  diagnosis. 

Prior  Work  Developing  a  Prototype  Stereoscopic  Digital  Mammography  System 

In  earlier  work,  we  developed  a  prototype  stereoscopic  digital  mammography  system 
which  consists  of  two  components:  the  first,  a  mammography  unit  modified  to  capture  stereo 
direct-digital  mammograms,  and  the  second,  a  workstation  to  display  the  stereo  mammograms. 

The  system  for  capturing  direct-digital  mammograms  was  developed  by  team  members  in 
the  Department  of  Radiology  at  the  University  of  Massachusetts  Medical  Center  (UMMC).  The 
prototype  laboratory  system  consists  of  a  standard  GE  Senographe,  Model  500TS,  as  the  x-ray 
source.  The  film  tray  of  the  unit  was  replaced  with  a  high  resolution  fluorescent  screen  to  which 
was  attached  an  experimental  CCD  (charge-coupled  device)  camera,  developed  by  Dr.  Andrew 
Karellas  (Karellas  et  al.,  1989, 1992a).  We  designed  and  built  a  special  mounting  system  to  hold 
the  compression  table,  fluorescent  screen  and  camera  fixed  while  the  gantry  and  x-ray  tube  were 
allowed  to  rotate,  enabling  the  capture  of  correctly  imaged  stereo  pairs. 

At  BBN,  we  developed  a  stereo  display  workstation  that  permits  the  radiologist  to  view  a 
pair  of  images  stereoscopically  and  to  control  various  aspects  of  their  presentation.  Left  and 
right  eye  views  are  presented  alternately  at  120  Hz,  in  non-interlaced  mode,  on  a  single,  high 
resolution  monitor.  The  viewer’s  two  eyes  are  alternately  occluded  in  synchrony  with  successive 
images  displayed  on  the  monitor  by  a  pair  of  StereoGraphics  liquid  crystal  shutters  mounted  as 
eyeglasses.  By  synchronizing  the  shuttering  of  the  glasses  to  the  display,  the  left  eye  always  sees 
the  appropriate  left-eye-image  and  the  right  eye  the  right-eye-image.  We  developed  software  that 
gives  the  user  control  over  the  brightness  and  contrast  of  the  displayed  stereo  image,  the  ability 
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to  move  a  cursor  in  depth  within  the  displayed  volume,  and  the  ability  to  invert  the  displayed 
grayscale  and  the  displayed  depth. 

Using  this  prototype  system,  we  collected  a  set  of  stereo  images  of  39  breast  biopsy 
specimens.  For  many  of  these  specimens,  we  were  able  to  obtain  multiple  images  spaced  apart  at 
2-degree  angular  increments.  From  those  multiple  images,  we  determined  that  the  optimal 
angular  separation  for  stereoscopic  viewing  of  the  specimen  tissue  was  about  6-degrees.  We 
expect  that  the  optimal  separation  for  stereo  views  of  the  compressed,  intact  breast  will  be  no 
greater  than  6-degrees,  and  possibly  slightly  less.  This  will  be  determined  from  cases  imaged 
early  in  this  project  and  then  fixed  for  the  remainder  of  the  project. 

We  also  conducted  a  diagnostic  reading  study  using  27  of  the  39  cases  for  which  we  had 
complete  materials,  including  digital  stereo  images  of  the  specimen,  localization  films,  specimen 
radiographs,  and  pathology  reports.  The  cases  were  read  blind,  in  a  random  order,  by  one  expert 
mammographer,  who  first  rated  the  probability  of  malignancy  after  viewing  the  two  orthogonal 
localization  mammograms,  and  then  re-rated  the  probability  of  malignancy  after  being  shown  the 
stereo  mammogram.  We  conducted  an  ROC  analysis  of  these  data  and  obtained  an  accuracy 
measure,  Az,  of  0.725  when  the  mammographer  viewed  the  standard  pair  of  nonstereo  views. 
Accuracy  increased  dramatically  to  0.939  when  the  stereo  mammogram  was  added. 
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BODY  OF  REPORT 


In  the  body  of  this  report  which  follows,  we  discuss  progress  during  Year  1  in  three  major 
areas;  (1)  improvements  made  to  the  stereo  display  workstation,  (2)  installation  and  testing  at 
UMMC  of  a  new  GE  full-field-of-view  digital  mammography,  and  its  first  use  in  acquiring  stereo 
mammograms  from  several  patients  prior  to  biopsy,  and  (3)  work  on  identification  of  visual 
features  of  lesions  and  normal  breast  tissue  which  can  be  seen  uniquely,  or  better,  in  depth. 

1.  Improvements  to  the  Stereo  Display  Workstation 

We  have  completed  a  substantial  amount  of  control  software  for  the  stereo  display 
workstation  during  this  first  year  of  the  project.  This  work  has  included  (1)  reorganization  of  the 
system’s  graphical  user  interface,  (2)  addition  of  new  functionality  to  give  the  user  control  over 
zooming,  panning  and  scrolling  of  the  stereo  image,  (3)  development  of  a  database  structure  for 
storing  the  stereo  images  and  parametric  display  information,  and  (4)  implementation  of  speech 
control  over  many  aspects  of  the  display  system.  We  discuss  each  of  these  developments  below. 

1 . 1  Reorganization  of  the  Graphical  User  Interface 

In  this  first  year,  we  have  begun  moving  from  an  interface  designed  with  the  needs  of  the 
developer  as  a  primary  concern  to  one  in  which  the  convenience  and  needs  of  the  mammographer 
as  user  take  center  stage.  In  particular,  we  have  moved  the  essential  controls  required  by  the 
mammographer  to  the  primary  window  for  immediate  accessibility,  and  moved  secondary 
controls,  needed  infrequently,  to  hidden  windows  accessed  from  items  on  the  menu  bar.  While 
we  will  continue  to  seek  to  make  the  graphical  interface  as  convenient  and  logical  as  possible  for 
the  mammographer,  we  expect  that  much  of  the  commonly-used  functionality  will  be  better 
implemented  using  speech  control  or  by  use  of  a  joystick,  thereby  avoiding  the  need  to  look  over 
at  the  system  control  monitor.  These  issues  are  discussed  at  greater  length  below. 

1.2  Addition  of  New  Control  Functionality  to  the  Stereo  Display  System 

We  have  added  zoom,  pan,  and  scroll  capabilities  to  the  stereo  display  during  Year  1. 
When  looking  at  tissue  containing  very  fine  detail,  such  as  a  dense  cluster  of  very  small 
microcalcifications,  we  have  found  that  it  is  very  helpful  to  be  able  to  zoom  in  on  the  finding  at 
either  2X  or  4X  magnification.  Although,  there  is  no  gain  in  absolute  information  in  a  zoomed 
image,  since  pixels  are  merely  replicated,  visual  perception  is  often  significantly  improved. 

Also,  when  a  stereo  image  is  magnified  by  zooming,  perceived  depth  is  magnified  by  the  same 
scale  factor.  Since  only  a  portion  of  the  original  image  is  visible  when  the  mammogram  is 
zoomed  (1/2  of  each  dimension  at  2X  and  1/4  at  4X),  the  ability  to  pan  and  scroll  within  the 
larger  available  image  becomes  important.  We  have  implemented  routines  that  allow  the  user  to 
carry  out  continuous  panning  and  scrolling  of  the  visible  display  window  within  the  larger  stored 
image. 
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As  we  begin  to  acquire  a  large  database  of  stereo  mammograms,  it  becomes  important 
that  we  develop  software  to  manage  this  database.  In  Year  1,  we  have  developed  capabilities  of 
storing  stereo  images  as  linked  pairs  which  define  a  case.  In  addition,  the  database  stores 
particular  information  about  each  case  that  allows  the  stereo  image  to  be  displayed  initially  with 
optimized  grayscale  windowing,  optimal  image  registration,  and,  optionally,  with  a  description  of 
the  case. 

1.3  Implementation  of  Speech  Control  over  the  Stereo  Display  System 

We  have  incorporated  speech  recognition  technology  into  the  stereo  display  workstation 
during  this  first  year.  This  technology  gives  the  user  control  over  most  aspects  of  the  stereo 
display  using  voice  commands.  A  primary  advantage  of  using  speech  to  control  the  display  is 
that  the  user  is  not  distracted  from  his/her  viewing  of  the  mammographic  images.  Without 
speech  control,  the  user  must  look  over  at  the  control  monitor  screen  and  use  a  mouse  or  the 
keyboard  to  manipulate  controls  displayed  on  the  monitor.  With  speech  control,  the  user  can 
command  changes  to  the  stereo  display  and  immediately  see  the  resulting  effects  on  the  stereo 
mammogram. 

The  speech  subsystem  includes  a  hardware  card  and  driver  software  developed  by  Speech 
Systems  Inc.  We  have  written  software  routines  and  speech  syntaxes  that  implement  a  range  of 
control  functions  by  speech.  The  user  wears  a  lightweight  headset  that  includes  a  boom 
microphone,  and  holds  a  small  control  incorporating  a  push-to-talk  button.  Although  the  system 
can  be  run  with  voice-activated  input,  we  have  found  that  use  of  the  push-to-talk  device  is 
preferable  since  it  allows  the  user  to  carry  on  other  conversations  intermixed  with  control 
commands. 

The  speech  commands  implemented  so  far  fall  into  four  categories:  (1)  case  selection,  (2) 
windowing  of  the  display  grayscale,  and  (3)  inversions  of  depth  and  grayscale,  and  (4)  3-D  cursor 
control.  The  full  set  of  speech  commands  is  given  in  the  Appendix.  Case  selection  commands 
allow  the  user  to  choose  a  particular  case  to  be  displayed.  Windowing  commands  permit  the  user 
to  slowly  (or  rapidly)  increase  or  decrease  image  brightness  or  contrast  and  to  halt  the  change 
when  an  appropriate  setting  is  reached.  The  system  will  also  allow  the  user  to  specify  particular 
brightness  or  contrast  settings,  and  to  save  current  settings  as  defaults  for  the  case.  Inversion 
coimnands  allow  the  user  to  invert  depth  in  the  displayed  mammogram  and  to  invert  grayscale. 
Inverting  depth  is  very  useful  in  bringing  tissue  at  the  back  of  displayed  image  to  the  foreground 
where  it  is  perceptually  more  salient.  Inverting  the  grayscale  causes  denser  tissue  to  be  displayed 
as  darker  rather  than  lighter.  This  gives  the  image  the  appearance  of  being  backlit  rather  than 
frontlit,  which  can  make  fine  detail  such  as  thin  spicules  easier  to  perceive.  The  cursor 
commands  turn  a  displayed  cursor  on  or  off,  and  determine  whether  it  can  change  its  position  in 
depth. 

1.4  Plans  for  Year  2 

In  Year  2,  we  plan  three  activities  related  to  further  refinement  of  the  stereo  display 
workstation.  The  first  concerns  an  investigation  of  a  three-dimensional  joystick  as  an  alternative 
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means  to  control  grayscale  windowing  (brightness  and  contrast),  to  control  movement  of  a  cursor 
in  the  displayed  volume,  and  to  control  panning  and  scrolling  of  the  visible  image  window  within 
the  stored  image.  It  is  likely  that  a  joystick  will  provide  a  user  with  a  more  natural  and  sensitive 
control  over  these  functions  than  either  speech  control  or  use  of  the  mouse.  In  the  current 
system,  grayscale  windowing  is  controlled  either  by  voice  commands  (e.g.,  “increase  brightness” 
or  “decrease  contrast  slowly”)  or  in  the  graphical  interface  by  moving  a  cursor  over  either  a 
contrast  or  a  brightness  button  and  using  the  two  mouse  buttons  to  control  changes.  The  joystick 
would  allow  brightness  and  contrast  to  be  controlled  by  separate  directions  of  movement.  We 
will  also  explore  use  of  the  joystick  to  control  positioning  of  a  cursor  in  the  displayed  volume, 
using  a  three-dimensional  joystick  to  permit  control  of  position  in  depth  as  well  as  in  the  frontal 
plane.  Finally,  we  will  examine  the  use  of  the  joystick  to  control  panning  and  scrolling  of  the 
visible  display  window  in  normal  and  magnified  images. 

The  second  activity  will  be  ongoing  refinement  of  the  user  interface.  As  we  gain 
experience  with  the  use  of  the  display  workstation  during  Year  2,  we  will  incorporate 
improvements  to  the  interface  based  on  comments  and  suggestions  made  by  mammographers 
using  the  system.  We  expect  that  the  most  efficient  and  natural  interface  will  involve  a 
combination  of  manual  and  speech  control,  with  categorical  or  discrete-state  commands  most 
naturally  assigned  to  speech  and  continuous,  graded  commands  controlled  manually  with  a 
joystick. 

The  third  activity  will  be  a  study  to  determine  the  feasibility  of  porting  the  stereo  display 
system  from  the  current  Matrox  Image- 1280  display  card,  and  supporting  Matrox  library,  to  a 
different  display  card  and  imaging  library.  The  Image- 1280  card,  with  a  modification  to  enable 
stereo  display,  is  no  longer  available  from  Matrox.  We  will  explore  other  means  of  displaying 
stereo  images  using  currently  available  imaging  cards,  one  of  several  available  image-processing 
libraries,  and  StereoGraphics  current  external  stereo-enabling  adapter.  The  goal  will  be  to 
configure  a  new  system  that  will  duplicate  the  current  system’s  functionality  with  minimal 
rewriting  of  our  software.  Our  interest  in  developing  this  alternative  display  workstation  is  both 
to  provide  a  backup  for  our  current  system  and  to  make  a  stereo  display  system  available  to  the 
mammographers  at  UMMC  where  the  stereo  images  are  being  acquired. 

2.  Installation  and  Testing  of  a  New  Digital  Mammography  Unit  at  UMMC 

The  original  mammographic  detector  for  stereoscopic  imaging  was  intended  to  be  a  small 
CCD-based  digital  imager  covering  an  area  of  approximately  6x6  cm.  This  sensor  would 
provide  relatively  high  contrast  and  spatial  resolution  in  a  confined  area  which  would  be 
adequate  for  the  stereoscopic  studies.  Although  we  have  made  very  significant  progress  with  this 
small  CCD  detector,  some  new  developments  have  prompted  us  to  make  an  adjustment  in  our 
plan  which  will  result  in  a  vast  improvement  in  our  technique.  Earlier  this  year  we  were  able  to 
install  a  full-field  digital  mammography  imager  (General  Electric)  which  is  integrated  with  a  GE 
DMR  x-ray  machine.  We  intend  now  to  make  use  of  this  new  device  to  acquire  stereo  images 
from  patients  enrolled  in  our  study. 
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2.1  Acceptance  Testing  of  the  New  Imaging  Unit 

Initial  testing  of  the  physical  characteristics  of  the  new  full-field  imager  have  revealed 
that  the  radiation  dose  with  this  device  can  be  even  lower  than  with  typical  small-field  CCDs, 
and  the  eontrast  resolution  and  overall  detectability  is  outstanding,  perhaps  the  best  that  we  have 
seen  with  any  digital  mammographic  device.  This  is  due  to  the  direct  contact  of  the  scintillator 
with  the  photosensitive  material  without  any  intermediate  fiberoptic  plate  or  taper. 

Measurements  of  the  modulation  transfer  function  with  this  imager  have  revealed  a  significantly 
higher  modulation  at  low  to  mid  frequencies  which  is  exactly  what  radiologists  need  to  enhance 
the  contrast  resolution.  This  objective  evaluation  was  followed  by  imaging  to  the  standard  ACR 
phantom,  and  the  overall  detectability  in  the  phantom  features  appears  to  be  significantly  better 
than  that  of  film  screen  systems.  It  appears,  therefore,  that  this  new  full-field  digital  device  is 
perfectly  capable  and,  in  fact,  nearly  ideal  for  stereoscopic  imaging. 

2.3  Modifications  and  Testing  for  Stereoscopic  Imaging 

We  have  proceeded  to  modify  the  system  by  enabling  it  to  take  exposures  from  0  to  +/- 
15  degrees  angulation  without  moving  the  compressed  breast  or  the  image  sensor.  Further,  we 
have  attached  a  high  precision  electronic  goniometer,  which  provides  a  digital  readout  of  the 
angle  in  better  than  0.5  degree  increments.  Moreover,  the  x-ray  collimation  ean  be  adjusted  to 
elose  in  to  the  area  of  interest  and  acquire  two  images  from  two  slightly  different  angles  for 
stereoscopic  fusion.  After  acquisition,  the  digital  images  are  sent  to  the  Radiology  Research 
Laboratory  where  they  are  either  written  on  a  CD  ROM  or  electronically  forwarded  to  BBN. 
Stereoscopic  images  of  specimens  already  obtained  with  this  new  system  suggest  a  dramatic 
improvement  in  image  quality  after  stereofusion  as  compared  to  our  original  images  with  the 
prototype  CCD  camera. 

2.4  Plans  for  Year  2 

This  new  system  has  been  installed  in  a  very  convenient  area  in  the  clinical  part  of  the 
Department  of  Radiology.  Acceptance  testing  of  the  unit  and  modifications  for  stereo  imaging 
are  complete.  In  the  past  several  days,  we  have  acquired  stereo  mammograms  from  the  first  four 
patients  enrolled  in  our  study,  prior  to  scheduled  breast  biopsies.  We  expect  to  enroll  and  acquire 
stereo  mammograms  on  approximately  125  patients  during  Year  2. 

3.  Identification  and  Development  of  Stereo-based  Visual  Features 

The  basic  task  has  been  to  refine  and  extend  our  present  list  of  2-D  diagnostic  features  to 
cover  features  newly  or  differently  visible  in  the  stereo  display.  The  original  plan  for  this 
beginning  stage  of  the  work  was  to  conduct  interviews  and  studies  with  three  expert 
mammographers,  but  that  presumed  the  availability  of  a  reasonably  large  number  of  stereo- 
imaged  cases.  We  still  intend  to  do  that  as  soon  as  possible  in  Year  2. 

The  limited  time  we  have  spent  on  feature  exploration  in  this  first  year  has  been  devoted 
to  examining  carefully  each  of  our  existing  set  of  specimen  cases,  as  well  as  new  ones  as  they 
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have  become  available,  manipulating  the  stereo  display  parameters  individually  and  in 
combination  to  bring  out  aspects  of  interest.  We  have  considered  it  important  to  do  each  of  these 
examinations  first  with  our  relatively  unschooled  eyes  to  bring  as  fresh  and  unbiased  a 
perspective  as  possible  to  the  exploration,  and  only  then  to  sit  with  one  of  our  expert 
mammographers,  first  to  review  our  findings  and  then  to  conduct  a  detailed  study  from  his 
trained  perspective.  This  has  allowed  us  to  develop  a  good  platform  for  the  full  activity  with  all 
three  of  our  experts  on  the  large  set  of  cases  in  Year  2. 

Though  our  exploration  to  date  has  been  limited,  it  has  been  quite  informative,  first  in 
clearly  confirming  the  promise  of  the  stereo  display,  and  second  in  providing  several  very 
interesting  specific  findings.  With  respect  to  the  general  promise  of  the  stereo  display,  we  have 
found  it  to  be  remarkably  sharp  and  clear,  and  to  convey  very  powerful  perceptions  of  the 
substance  and  structure  of  the  lesions  in  depth.  There  is  much  that  is  seeable  and  much  to  be 
visually  explored  in  these  richly  structured  scenes. 

3.1  Stereo-based  Features  of  Masses 

We  are  not  in  a  position  at  this  point  to  offer  anything  of  an  overview  of  what  the  display 
can  provide  with  respect  to  specific  new  features  .  It  will  take  the  study  of  the  larger  set  of 
specimens  in  Year  2  to  develop  one.  What  we  can  report  now  is  our  discovery  of  several 
features,  particularly  of  masses,  that  we  think  are  indicative  of  much  more  to  come  The  first  of 
these  is  the  discovery  in  one  of  the  lesions  of  very  clear  nodules  at  the  tips  and  at  intervals  along 
the  length  of  many  of  its  spicules.  To  our  knowledge,  this  feature  has  not  previously  been 
reported,  possibly  because  it  has  never  been  discernible  in  standard  2-D  mammographic 
displays  or  in  standard  pathology  preparations.  The  second  discovery  in  another  of  the 
specimens  is  the  apparent  proclivity  of  its  spicules  to  emanate  from  the  center  of  the  lesion,  and 
not,  as  generally  appears  to  be  the  case  in  2-D  silhouettes  of  masses,  to  grow  out  from  the 
surface.  It  suggests  that  some  lesions  that  look  like  masses  in  2-D  might  actually  be  very  dense 
forms  of  architectural  distortion.  The  stereo  display  enables  tracing  the  paths  of  individual 
spicules  from  end  to  end,  and  it  is  clear  in  this  particular  specimen  that  they  emanate  from  a  point 
in  the  middle  of  the  lesion.  Another  striking  finding,  with  a  specimen  showing  a  cluster  of  small 
masses,  is  how  the  spicules  bridge  the  masses,  as  if  one  mass  spawned  the  next  as  an  outgrowth 
of  one  of  its  spicules.  It  suggests  to  us  that  the  nodes  visible  on  the  spicule  in  the  case  mentioned 
above,  though  not  visible  here,  might  be  the  sites  of  such  proliferation.  These,  of  course,  are  very 
loose  speculations,  but  they  nicely  illustrate  the  richness  of  what  can  be  seen  and  the  potential 
for  new  information  and  insights  to  be  gained  with  the  stereo  display. 

3.2  Stereo-based  Features  of  Calcification  Clusters 

The  few  cases  we  have  studied  where  the  lesion  is  seen  as  a  cluster  of  calcifications  have 
also  been  revealing,  mainly  of  the  richness  of  information  that  is  conveyed.  The  stereo  display 
enables  seeing  the  calcifications  arrayed  in  distinctly  separate  groups,  some  obviously  following 
ductal  branches.  The  branches  can  be  seen  to  go  in  different  directions.  Individual  calcifications 
that  in  a  conventional  2-D  display  would  be  seen  all  in  one  undifferentiated  group  can  be 
perceptually  segregated  in  depth  into  distinct  sub-groups  and  branches,  each  with  its  own 
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distinguishable  size,  shape  and  orientation.  Moreover,  the  global  pattern  of  the  subgroups  in 
combination  can  be  perceived,  potentially  revealing  the  ductal  structure  that  has  been  occupied 
by  the  lesion.  Several  new  and  potentially  diagnostic  features  can  probably  be  farmed  from  this 
elaborate  new  information,  e.g.,  to  reveal  the  anatomical  locus  of  the  lesion  in  the  parenchyma 
and  how  it  has  insinuated  itself  there. 

3.3  Plans  for  Year  2 

This  limited  exploratory  effort  has  provided  strong  verification  that  the  stereo  display  will 
provide  a  rich  new  set  of  diagnostic  features.  It  has  also  enabled  us  to  become  familiar  enough 
with  the  display,  and  with  the  domain  of  the  new  features  to  proceed  at  an  accelerated  pace  at  the 
remainder  of  this  work  in  Year  2. 

As  soon  as  we  have  accumulated  a  sufficiently  large  number  of  cases  to  begin  it,  and  we 
expect  that  to  be  the  case  within  two  or  three  months,  we  will  pursue  the  originally  planned 
approach  of  systematic  review  and  study  with  our  three  expert  mammographers.  We  anticipate 
that  we  can  conduct  the  consensus  meeting  that  will  produce  the  final  list  of  new  stereo  features 
by  Month  20.  We  expect  that  we  can  then  accelerate  development  of  the  feature  checklist  by 
Month  24. 

4.  Presentations  of  Research  During  Year  1 

At  the  invitation  of  Dr.  Susan  Blumenthal,  Deputy  Assistant  Secretary  for  Health 
(Women’s  Health),  Dr.  Getty  participated  in  a  meeting,  the  Federal  Technology  Transfer 
Workshop  on  Breast  Cancer  Detection,  Diagnosis,  and  Treatment,  held  in  Washington  on  May 
1-2,  1997.  He  presented  the  research  efforts  of  this  project  to  the  workshop,  in  a  talk  entitled 
“Stereoseopic  Digital  Mammography:  Improving  Detection  and  Diagnosis  of  Breast  Cancer.” 

Dr.  Getty  also  presented  a  demonstration  of  stereoscopic  mammography  to  individual 
participants,  in  the  form  of  sample  stereo  mammograms  captured  on  35mm  slides  and  viewed  in 
handheld  stereoscopic  viewers. 

5.  Schedule  and  Level  of  Effort 

The  project  ran  at  a  level  of  effort  and  expenditure  through  Year  1  of  close  to  100%  of 
budget.  Because  of  the  delays  in  acquisition  and  installation  of  the  GE  full-field-of-view  digital 
mammography  unit,  we  are  behind  our  original  schedule  in  accrual  of  imaged  cases.  However, 
we  are  confident  that  we  will  catch  up  in  case  acquisition  during  Year  2.  All  staff  and 
consultants  continue  with  the  project  as  planned. 


13 


CONCLUSIONS 


All  of  the  work  proposed  for  the  first  year  to  improve  the  Stereo  Display  Workstation  has 
been  completed,  including:  reorganization  of  the  graphical  user  interface;  addition  of  new 
functionality  to  the  stereo  display  system;  and  implementation  of  speech  control  over  the  stereo 
display  station.  Installation  and  acceptance  testing  of  the  GE  full-field-of-view  Digital 
Mammography  Unit  is  now  complete.  Delay  in  delivery  of  the  unit  from  the  manufacturer  put 
this  work  behind  schedule,  and  caused  attendant  delays  in  the  collection  of  case  images  and,  in 
turn,  a  slowing  of  the  feature  development  work.  We  have,  however,  been  able  to  accomplish 
much  of  the  feature  development  work  planned  for  the  first  year  by  working  intensively  with 
available  specimen  images,  and  we  expect  that  we  can,  therefore,  accelerate  the  feature 
development  work  as  cases  are  accrued  with  the  new  equipment.  This  optimistic  projection  is 
also  justified  by  the  fact  that  the  pool  of  patients  available  for  imaging  has  recently  been  greatly 
expanded  due  to  the  increased  utilization  of  core  biopsies  at  UMMC. 

We  can  draw  two  general  conclusions  from  this  first  year  of  work.  The  first  is  that  the 
stereo  display  is  all  that  we  had  expected  and  hoped  it  would  be,  and  more.  It  provides  displays 
that  are  easy  to  read  and  highly  informative  -  sharp  images  of  the  fine  features  of  lesions  that 
have  to  be  discriminated  for  diagnosis,  and  all  perceived  with  remarkably  strong  impressions  of 
position,  separation  and  orientation  in  depth.  Our  initial  explorations  suggest  that  the  display 
will  provide  rich  new  sources  of  diagnostic  information.  The  second  general  conclusion  is  that 
the  various  aspects  of  development  of  the  system  we  initially  proposed  to  enable  easy  and 
versatile  access  to,  and  interaction  with,  the  display  system  parameters  were  attainable,  and  most 
have  proven  to  be  quite  necessary.  We  have  found  that  to  bring  out  the  full  richness  and  clarity 
of  the  display  requires  sensitive  adjustment  of  the  various  parameters,  and  that  it  is  important  to 
make  those  interactions  as  natural  and  convenient  as  possible.  Those  interactive  capabilities  we 
initially  proposed  have  been  accomplished  to  very  good  and  useful  effect,  and  we  plan  further 
refinements  for  Year  2.  All  in  all,  this  project  is  proving  to  be  very  productive  and  suggestive 
that  some  truly  large  gains  in  diagnostic  accuracy  will  be  obtained. 
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APPENDIX 


Description  of  the  Stereo  Display  System 

The  Stereo  Display  System  displays  a  pair  of  digitized  x-ray  images  stereoscopically  and 
provides  the  user  with  control  over  various  aspects  of  the  display  through  both  a  graphical 
and  speech  interface.  Most  commands  can  be  invoked  both  through  a  menu  item  and/or 
button  on  the  graphical  user  interface  (GUI)  or  through  a  voice  command. 

When  the  GUI  is  used,  the  graphical  system  by  its  nature  provides  visual  feedback  about 
which  command  the  user  selected.  For  voice  commands,  the  system  provides  auditory 
feedback  through  a  headset  about  which  command  it  will  execute  based  on  what  it  thinks 
the  user  said.  Except  for  the  “Save”  command,  the  user  is  not  required  to  confirm  the 
command  before  it  is  executed. 

The  commands  fall  into  five  general  categories: 

1 .  Case  selection 

2.  Grayscale  manipulation 

3.  Image  display 

4.  Cursor  control 

5.  Speaker  gender 

This  appendix  describes  the  commands  in  each  of  the  categories  and  how  they  are 
implemented  in  the  GUI  and  VOICE  interfaces. 


Conventions 

The  following  conventions  are  used  in  describing  the  VOICE  commands: 

1.  Words  in  parentheses  ()  are  optional  and  are  included  for  naturalness  or  to  improve 
recognition. 

Example:  “Report  (settings)” 

Here  the  user  can  say  either  “Report”  or  “Report  settings”. 

2.  Words  or  phrases  separated  by  commas  in  square  brackets  indicate  alternative  items 
that  can  be  said  at  that  point  in  the  text. 

Example:  “Invert  [grayscale,  depth]” 

The  user  can  say  either  “Invert  grayscale”  or  “Invert  depth”. 

The  user  should  not  say  “Invert”. 

3.  Italicized  words  in  angle  brackets  indicate  a  range  of  items  that  can  be  spoken  at  that 
point  in  the  text. 

Example:  Case  (number)  <a  number  from  1  to  299>  “ 

The  user  can  follow  either  “Case”  or  “Case  number”  with  a  number  between  1  and 
299. 
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stereo  Display  Windows 

The  Stereo  Display  application  has  a  startup  window  whose  function  is  to  call  the 
initialization  routines  for  the  speech  and  imaging  functions  of  the  system  when  the  user  is 
ready  to  begin.  When  the  initialization  routines  finish,  this  window  then  invokes  the 
main  window  of  the  system. 

In  addition  to  the  startup  window,  the  Stereo  Display  application  has  four  other  windows: 

•  The  main  window. 

This  window  is  the  primary  window  of  the  application.  From  this  window  the  user 
selects  cases  and  sees  information  about  the  case  currently  being  viewed.  It  has 
control  buttons  for  selecting  the  type  of  display  and  for  invoking  the  cursor.  It  also 
has  the  menu  bar  for  accessing  all  of  the  functionality  and  the  other  windows  of  the 
system. 

•  The  Enhancements  Palette  window. 

This  window  contains  the  controls  for  manipulating  the  grayscale  map  as  well  as 
inverting  depth  or  color.  The  Registration  window  can  also  be  called  from  this 
window. .  This  window  is  invoked  from  the  Tools->Enhancements  Palette  menu  on 
the  main  window. 

•  The  Registration  window. 

This  window  contains  the  controls  for  aligning  the  images,  for  panning/scrolling,  and 
for  magnifying.  This  window  is  invoked  from  the  Tools->Registration  menu  on  the 
main  window  or  from  the  “Adjust  Registration”  button  on  the  Enhancements  Palette 
window. 

•  The  Cursor  window. 

This  window  is  invoked  when  the  user  requests  a  cursor  on  the  display  by  clicking  the 
Stereo  Cursor  icon.  It  uses  the  X  and  Y  coordinates  of  the  mouse  cursor  within  its 
borders  to  position  the  crosshair  cursor  on  the  display. 

1 .  Case  Selection 

A  case  is  comprised  of  two  x-ray  image  files:  one  image  file  is  intended  to  be  seen  by  the 
left  eye  to  see  and  the  other  by  the  right  eye.  A  case  can  be  selected  for  viewing  by  either 
explicitly  selecting  from  disk  the  two  image  files  associated  with  the  case,  or  by  selecting 
a  case  from  a  database. 

Read  Case  Files  from  a  Disk 

This  command  allows  the  user  to  specify  the  two  image  files  for  viewing.  The  image  on 
the  stereo  screen  shows  the  data  as  they  are  in  the  raw  image  files,  without  any 
enhancements. 

GUI  Controls:  Load  Stereo  Pair  button,  File->Load  Stereo  Pair  menu 

This  command  brings  up  a  custom  file  selection  dialog  that  allows  the  user  to  specify  the 
two  image  files  at  once.  The  first  file  selected  is  seen  by  the  left  eye  and  the  second  file  is 
seen  by  the  right  eye. 
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VOICE:  (not  implemented) 


Select  a  Case  from  a  Database 

The  user  can  select  a  case  from  a  database  of  cases  viewed  previously  by  the  user  and 
saved.  When  the  user  selects  a  case  from  the  database,  the  system  loads  the  two  image 
files  and  initializes  the  display  to  the  default  settings  that  were  saved  with  the  case.  On 
the  main  window,  the  system  displays  the  case  number,  the  patient  ID,  the  case 
description,  and  the  names  of  the  images  files  that  the  user  is  viewing.  All  of  this 
information  was  obtained  from  the  database  when  the  case  was  selected. 

GUI  Controls:  Next  Case  button,  Case->Next  Case  menu 

This  command  brings  up  a  dialog  window  with  the  database  of  cases  shown  in  a  table. 
The  user  selects  a  case  by  clicking  on  the  row  that  describes  the  case.  When  the  user 
clicks  the  OK  button,  the  case  is  displayed. 

VOICE  command  sequence: 

User:  “Next  (case)” 

System:  After  the  tone,  please  say  “Case”  followed  by  the  number.  <beep> 

User:  “Case  <a  number  from  1  to  299  >  “ 

System:  Case  <number>  is  Specimen  C. 

User:  “OK”,  or  “Cancel”,  or  another  case  number. 

The  dialog  above  between  the  user  and  system  demonstrates  how  a  user  would  select  a 
case  from  the  database  using  voice  commands.  After  the  system  prompts  for  a  case 
number,  the  same  dialog  window  as  invoked  by  the  GUI  control  is  displayed  so  that  the 
user  can  view  the  database  while  making  a  choice. 

Save  a  case  to  the  Database 

The  system  allows  the  user  to  build  a  database  from  cases  that  were  loaded  initially  by 
reading  the  files  directly  from  disk  (see  the  method  above).  After  loading  a  case,  the  user 
can  optimize  the  image  by  changing  grayscale  settings  and  registration.  When  the  image 
is  satisfactory,  the  user  can  then  save  the  case  under  a  unique  case  ID  number.  This 
command  requires  the  user  to  confirm  the  “save”  because  this  action  will  overwrite  data 
previously  stored  for  this  case. 

Saving  a  case  writes  the  following  information  about  that  case  to  the  database: 

•  a  unique  ID  number  assigned  by  the  system 

•  a  patient  ID  field  (text  from  the  Patient  data  entry  field  on  the  main  window) 

•  the  name  of  the  image  file  to  present  to  the  left  eye 

•  the  name  of  the  image  file  to  present  to  the  right  eye 

•  the  current  brightness  setting 

•  the  current  contrast  setting 

•  a  descriptive  text  field  (text  from  the  Description  data  entry  field  on  the  main 
window) 

•  the  X  and  Y  offset  for  the  left  eye  image 
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•  the  X  and  Y  offset  for  the  right  eye  image 

After  a  case  has  been  saved,  the  user  can  change  the  settings  or  descriptions  in  the 
database  for  that  case  by  first  selecting  the  case,  then  changing  settings  or  typing  in  the 
Patient  or  Description  field  on  the  main  window,  and  finally  “saving”  the  case  again. 
The  new  data  replaces  the  old  data  in  the  database. 

GUI  Controls:  Case->Save  Case  menu 

This  command  brings  up  a  confirmation  dialog  that  asks  the  user  to  confirm  the  action. 

VOICE  command  sequence: 

User:  “Save  (case)” 

System:  Yes  or  No.  <beep> 

User:  “Yes  “  or  “No” 

System:  Case  saved,  or  Not  saved. 


2.  Grayscale  Manipulation 

Before  they  are  displayed  on  the  screen,  the  8-bit  data  values  in  an  image  file  pass 
through  a  look-up  table  that  maps  each  of  the  input  data  values  into  an  output  display 
value  between  0  (full  white)  and  255  (full  black).  By  default,  the  table  maps  each  input 
data  value  onto  itself.  The  resulting  picture  on  the  display  is  a  true  representation  of  the 
image  data  as  it  exists  in  the  file.  The  user  can  change  the  mapping  of  input-to-output 
values  to  enhance  the  appearance  of  the  picture  on  the  screen.  By  changing  the  mapping 
the  user  can  change  visual  aspects  of  the  display  such  as  brightness  and  contrast,  create  a 
“negative”  of  the  image,  or  highlight  a  specific  data  value. 

Grayscale  manipulation  on  the  GUI  is  performed  from  the  Enhancements  Palette 
window.  All  the  controls  mentioned  in  this  section  are  on  the  Enhancements  Palette 
window 

Increase/Decrease  Brightness 

This  command  increases  or  decreases  the  overall  brightness  level  of  the  image  from  its 
current  setting.  The  value  of  the  brightness  level  ranges  from  0  to  255.  The  user  can 
change  this  value  in  large  or  small  increments  as  needed.  For  the  default  linear  map,  the 
higher  the  value  the  brighter  the  display  (if  the  map  is  inverted,  the  higher  the  value  the 
darker  the  image). 

GUI  Controls:  Scrollbar  attached  to  the  “Brightness”  rectangle 
Rectangle  labeled  “Brightness” 

These  controls  allow  the  user  to  make  incremental  or  gross  changes  to  the  brightness 
value. 
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With  the  scrollbar,  the  user  can  make  small  changes  to  the  value  by  clicking  the  left  or 
right  arrows,  or  make  larger  changes  by  dragging  the  slider  or  clicking  in  the.  With  the 
Brightness  rectangle,  the  user  can  increase  or  decrease  the  value  continuously  by  placing 
the  cursor  in  the  rectangle  then  holding  down  either  the  right  or  left  mouse  button 
respectively.  Releasing  the  mouse  button  stops  the  change. 

VOICE  commands:: 

“Increase  brightness  [slowly]” 

“Decrease  brightness  [slowly]” 

“Stop” 

When  the  user  gives  the  “Increase  or  Decrease  brightness  command,  the  brightness  level 
of  the  image  begins  to  change  quickly  in  the  designated  direction.  If  the  word  [slowly]  is 
added,  the  level  changes  by  one  every  second.  The  level  continues  to  change  until  the 
user  says  “Stop”  or  until  an  end  value  (0  or  255)  is  reached.  In  addition,  this  command 
also  updates  the  GUI  controls  to  reflect  the  new  value. 

Increase/Decrease  Contrast 

This  command  increases  or  decreases  the  range-of-contrast  value.  The  contrast  range 
value  is  a  number  between  0  and  255  that  defines  the  width  of  a  window  about  the 
brightness  level.  Data  values  in  this  window  will  be  mapped  from  white  (0)  to  black  ( 
255).  All  data  values  below  this  range  will  be  mapped  to  white  and  all  data  values  above 
this  range  will  be  mapped  to  black.  Increasing  the  contrast  range  decreases  the  amount  of 
contrast  in  the  display:  conversely,  decreasing  the  contrast  range  increases  the  amount  of 
contrast  in  the  image. 


GUI  Controls:  Scrollbar  attached  to  the  “Contrast”  rectangle 
Rectangle  labeled  “Contrast” 

VOICE  commands:: 

“Increase  contrast  [slowly]” 

“Decrease  contrast  [slowly]” 

“Stop” 

These  commands  are  implemented  in  the  same  manner  as  the  Brightness  level  controls 
and  commands.  (See  Increase/Decrease  Brightness  section  presented  earlier  more 
information.) 

Set  Brightness  to  a  Specific  Vaiue 

This  command  sets  the  brightness  level  to  a  value  between  0  and  255.  The  higher  the 
value  the  brighter  the  display.  If  the  linear  map  is  inverted,  the  higher  the  value  the 
darker  the  image. 

GUI  Controls:  Brightness  Scrollbar 

Rectangle  labeled  “Brightness” 
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The  same  controls  for  increasing/decreasing  brightness  are  used  to  set  the  brightness  to  a 
specific  value.  (See  the  Increase/Decrease  Brightness  section  described  earlier  for  how  to 
use  the  controls).  The  user  operates  the  GUI  controls  until  the  desired  value  appears  in 
the  text  field  to  the  right  of  the  “Brightness”  rectangle. 

VOICE  commands: 

“Set  brightness  to  <a  number  from  0  to  255>  “ 

This  command  sets  the  brightness  level  to  the  number  specified.  The  display  is  changed 
to  reflect  the  new  brightness  value.  The  GUI  controls  are  also  reset  to  the  new  value. 

Set  Contrast  to  a  Specific  Vaiue 

This  command  sets  the  range-of-contrast  value  to  a  number  between  0  and  255:  the 
larger  the  number  the  less  the  amount  of  contrast  in  the  display  and  vice  versa. 

GUI  Controls:  Scrollbar  attaches  to  the  “Contrast”  rectangle 
Rectangle  labeled  “Contrast” 

The  same  controls  for  increasing/decreasing  contrast  are  used  to  set  the  contrast  to  a 
specific  value.  (See  the  Increase/Decrease  Brightness  section  described  earlier  for  how  to 
use  the  controls).  The  user  operates  the  controls  until  the  desired  value  appears  in  the  text 
field  to  the  right  of  the  “Contrast”  rectangle. 

VOICE  commands: 

“Set  contrast  to  <a  number  from  0  to  255>  “ 

This  command  sets  the  range-of-contrast  value  to  the  number  specified.  The  display  is 
changed  to  reflect  the  new  range  value.  The  GUI  controls  are  also  reset  to  the  new  value. 


invert  Grayscale 

This  toggle  command  inverts  the  linear  grayscale  mapping.  Normally,  input  data  are 
mapped  linearly  into  output  values  ranging  from  pure  white  (0)  to  pure  black  (255). 
Inverting  the  map  causes  the  output  values  for  the  same  input  values  to  range  from  black- 
to-white  instead.  This  has  the  effect  of  producing  a  “negative”  of  the  image.  Inverting 
the  grayscale  again  returns  the  mapping  and  display  to  its  former  state. 

GUI  Controls:  Invert  GRAYSCALE  button 

VOICE  commands: 

“Invert  [grayscale/  black  and  white]  “ 

Reset  Grayscale 

This  command  resets  the  brightness  level  and  contrast  range  to  their  default  values  of  128 
and  256  respectively. 
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GUI  Controls:  Reset  button 


VOICE  commands: 

“Reset  (grayscale)  “ 

Both  commands  cause  the  controls  on  the  GUI  to  be  reset  to  the  default  values.  The 
display  reflects  the  new  settings. 

Highlight  a  Data  Value 

This  command  initially  sets  the  contrast  range  to  one  and  then  displays  the  input  data 
value  that  maps  to  the  brightness  level  as  pure  white,  thus  highlighting  that  one  data 
value.  The  remainder  of  the  data  are  mapped  linearly  into  values  between  0  and  126. 
This  allows  the  highlighted  value  to  be  observed  against  a  dimmed  version  of  the  image. 

GUI  Controls:  Highlight  check  box 

VOICE  commands: 

“Highlight  [On/Off]  “ 

When  highlighting  is  enabled,  the  image  is  set  to  half  of  its  full  brightness  and  only  the 
data  whose  value  equals  the  current  brightness  level  is  mapped  into  pure  white.  The  user 
can  highlight  single  data  values  by  leaving  the  contrast  range  at  one  and  changing  the 
brightness  level,  or  highlight  a  range  of  values  about  the  brightness  level  by  changing  the 
contrast  range.  Turning  highlighting  off  returns  the  image  to  the  visual  state  prior  to  the 
“highlight  on”  command. 

Report  Grayscale  Settings 

This  command  enables  the  user  to  hear  the  exact  values  of  the  current  brightness  and 
contrast  settings. 

GUI  Controls:  (N/A)  Settings  are  viewable  on  the  Enhancements  Palette  window. 

VOICE  command  sequence: 

User:  “Report  (settings)  “ 

System:  Reporting. ..Brightness  120,  Contrast  46 

Following  this  command,  the  user  hears  a  spoken  report  of  the  current  brightness  and 
contrast  settings  in  the  headset. 

Save  Grayscale  Settings 

The  “Save”  command  described  earlier  allows  the  user  to  save  the  current  brightness  and 
contrast  settings  for  a  case  so  that  when  the  case  is  recalled  at  a  later  time,  it  will  appear 
as  it  did  when  it  was  saved.  (See  the  Save  command  for  more  information.) 
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Image  Display 
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Request  Stereo  or  Mono  Display 

The  user  can  choose  a  stereo  or  a  monaural  view  of  a  case.  In  the  stereo  display,  the 
perception  of  a  three-dimensional  image  is  achieved  by  presenting  one  image  to  the  left 
eye,  while  presenting  a  different  image,  acquired  from  a  slightly  different  point-of-view, 
to  the  right  eye.  The  human  visual  system  is  able  to  fuse  these  two  images  into  one 
containing  depth. 

A  mono  display  simply  presents  the  same  image  (in  this  case  the  image  presented  to  the 
left  eye)  to  both  eyes.  The  perceived  image  has  no  depth. 

GUI  Controls:  Stereo  button.  Mono  button,  Display->Stereo  menu,  Display->Mono 
menu 

VOICE  commands: 

“Stereo  (display)” 

“Mono  (display)” 


Adjust  Alignment 

This  capability  allows  the  user  to  move  each  of  the  images  up,  down,  left  or  right 
independently  to  correct  for  any  misalignment  of  the  images  that  may  have  occurred 
when  the  pictures  were  taken. 

The  alignment  controls  are  on  the  Registration  window 

GUI  Controls:  Scrollbar  attached  to  a  rectangle  identified  by  eye  and  direction 
Rectangle  identified  by  eye  and  direction 

Each  combination  of  eye  and  direction  (left/horizontal,  left/vertical,  right/horizontal, 
right/vertical)  has  two  controls,  similar  to  the  controls  used  for  brightness  and  contrast. 
With  these  controls,  the  user  can  move  each  image  independently  in  the  horizontal  (left 
and  right)  or  vertical  (up  and  down)  direction. 

To  operate  any  of  the  four  scrollbars,  the  user  clicks  the  left  or  right  arrows  to  move  the 
image  by  small  increments,  or  drags  the  slide  or  clicks  in  the  track  to  move  in  larger 
increments.  To  use  any  of  the  eye/direction  rectangles,  the  user  places  the  cursor  in  the 
rectangle  then  holds  down  either  the  left  or  right  mouse  button  to  move  the  image 
continuously  in  one  direction  (left  or  up)  or  the  opposite  direction  (right  or  down) 
respectively.  Releasing  the  mouse  button  stops  the  movement. 

Once  the  images  are  aligned,  the  user  can  save  these  offsets  while  viewing  this  case  by 
clicking  the  OK/Save  button  when  exiting  the  window.  If  the  user  clicks  Cancel  to  exit 
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the  window,  the  display  is  reset  to  the  settings  in  effect  prior  to  invoking  the  Registration 
window. 

VOICE  commands:  (not  implemented) 


Pan  or  Scroll  Image 

This  command  allows  the  user  to  move  the  image  on  the  display  up  or  down  (scrolling), 
or  left  or  right  (panning).  In  this  case  the  left  and  right  eye  images  move  together. 

This  control  is  on  the  Registration  window. 

GUI  Controls:  Scroll  check  box 

To  enable  panning  or  scrolling,  the  user  clicks  the  pan/scroll  check  box  to  write  an  “X”  in 
the  box.  To  disable  movement,  the  user  clicks  in  the  check  box  again  to  remove  the  “X”. 
When  panning  and  scrolling  are  enabled,  the  images  are  coupled  together  and  the 
alignment  controls  for  the  left  eye  serve  to  move  the  image  on  the  display  up,  down,  left, 
or  right.  The  alignment  controls  for  the  right  eye  are  disabled. 

When  panning  and  scrolling  are  disabled,  the  images  are  uncoupled,  the  alignment 
controls  for  the  right  eye  are  enabled,  and  the  user  can  again  move  each  image 
independently. 

VOICE  commands:  (not  implemented) 

Zoom  Image  IX,  2X,  or  4X 

The  user  can  enlarge  the  image  on  the  display  by  a  factor  of  1,  2  or  4.  The  image  is 
magnified  about  the  center  of  the  current  view. 

This  control  is  on  the  Registration  window. 

GUI  Controls:  IX  radio  button,  2X  radio  button,  4X  radio  button 

The  user  can  select  the  magnification  factor  by  clicking  one  of  the  radio  buttons. 

VOICE  commands:  (not  implemented) 

Invert  Depth 

This  toggle  command  switches  the  image  presentation  to  each  eye  (i.e.,  the  image 
presented  to  the  left  eye  is  shown  to  the  right  eye  instead  and  vice  versa).  The  effect  is  an 
inversion  in  depth;  whatever  appears  to  have  been  at  the  back  of  the  display  now  appears 
in  front  and  what  was  in  front  appears  to  have  moved  to  the  back. 

This  control  is  on  the  Enhancements  Palette  window. 
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GUI  Controls:  Invert  DEPTH  button 

VOICE  commands: 

“Invert  depth” 

Cursor 

Display  a  Crosshair  Cursor 

Move  the  Crosshair  Cursor  within  a  Depth  Plane 

Sometimes  the  user  might  want  to  point  to  some  interesting  feature  in  the  image.  This 
command  provides  a  large,  crosshair  cursor  on  the  display  that  the  user  can  position 
horizontally,  vertically,  or  in  depth  with  the  aid  of  the  mouse. 

GUI  Controls:  Stereo  Cursor  Crosshair  Icon  on  the  main  window 

To  enable  the  use  of  the  cursor  (cursor  mode),  the  user  clicks  on  the  Crosshair  icon.  The 
system  responds  by  bringing  up  the  Cursor  window.  This  window  covers  the  entire 
console  screen;  it’s  purpose  is  to  use  the  location  of  the  mouse  cursor  in  its  window  to 
position  the  crosshair  cursor  on  the  display.  Now  when  the  user  moves  the  mouse,  that 
movement  is  translated  into  a  corresponding  movement  of  the  crosshair  cursor  on  the 
display.  This  X/Y  movement  takes  places  in  whatever  depth  plane  the  cursor  is  in. 

To  exit  cursor  mode  and  return  to  the  main  window,  the  user  clicks  Close  on  the  menu 
bar. 

VOICE  commands: 

“Cursor  On” 

“Cursor  Off’ 

The  voice  command  “Cursor  On”  enables  cursor  mode  and  has  the  same  effect  as 
clicking  on  the  GUI  control. 

The  voice  command  “Cursor  Off’  disables  cursor  mode  and  is  comparable  to  clicking 
Close  on  the  Cursor  window. 


Move  the  Crosshair  Cursor  to  Different  Depth  Planes 

In  addition  to  moving  horizontally  and  vertically,  the  crosshair  cursor  can  also  move 
forward  and  backward  in  depth,  giving  the  appearance  of  moving  in  and  out  of  the  image 
itself.  Once  the  user  positions  the  cursor  in  depth,  then  moving  the  mouse  moves  the 
cursor  horizontally  and  vertically  in  that  depth  plane. 

NOTE:  Before  voice  was  added,  this  function  was  implemented  by  holding  down  the  left 
and  right  mouse  buttons  to  move  backward  and  forward  respectively.  However,  the 
speech  system  uses  the  right  mouse  button  exclusively  as  the  push-to-talk  device  when 
speech  input  is  enabled.  Consequently,  in  order  to  continue  to  use  the  right  mouse  button 
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as  a  control,  both  the  GUI  and  voice  commands  below  disable  voice  so  that  the  right 
mouse  button  can  be  used  as  a  controlling  device. 

GUI  Controls:  Move  in  Depth->Enable  menu,  Move  in  Depth->Disable  menu 

VOICE  commands: 

“Cursor  Move” 

These  commands  disable  speech  input  and  enable  the  ability  to  move  the  cursor  in  depth 
using  the  mouse  buttons.  Once  speech  input  is  disabled,  the  user  can  hold  down  the  left 
and  right  mouse  buttons  to  move  the  cursor  backward  and  forward  respectively.  The 
user  cannot  issue  any  voice  commands  during  this  time,  however.  When  the  user  wants 
to  return  to  voice  mode,  the  user  either  selects  the  Move-in-Depth->Disable  menu  or 
double  clieks  either  of  the  mouse  buttons.  These  actions  end  the  ability  to  move  the 
cursor  in  depth  and  re-enable  the  ability  to  issue  voice  commands.  The  cursor  remains  at 
whatever  depth  plane  it  resided  in  when  movement  in  depth  was  disabled. 
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